Reduce the First Token Latency by 3.25 Times: Xiaohongshu Collaborates with Peking University and Shanghai Jiao Tong University to Propose HYPIC, Equipping Hybrid Attention Large Models with Location-Independent Caching

The main battlefield of large model services is shifting toward retrieval-augmented question answering, multi-document summarization, and long-range agents. The request prompt is composed of dozens to hundreds of semantically independent segments (retrieved documents, skills explanations, memory, historical rounds), forming ultra-long context with tens of thousands to hundreds of thousands of tokens. The pre-filling stage dominates the computing costs, becoming the most prominent cost source for service providers, and triggering more challenging problems.